Asynchronous COMID: the theoretic basis for transmitted data sparsification tricks on Parameter Server

Asynchronous FTRL and L2 norm done at server are two widely used tricks to improve training efficiency, but their convergences are not well-proved. In this paper, we propose asynchronous COMID algorithm and prove its convergence. Then, we establish the equivalence between asynchronous COMID and the above two tricks. Thus, the convergences of above two tricks are also proved. Experimental results show asynchronous COMID reduces the burden of the network without any harm on the convergence speed and final output. INTRODUCTION There are a lot of tricks in machine learning application to get higher training efficiency, better classification accuracy and the ability of solving unconvex optimization. Some of them are reasonable and well-proved, like setting a better initial model parameters to reduce training time. But most of other tricks are lack of proof. Most of those tricks can only be used suitably depending on users’ experience, like deciding the size of batch and constructing a DNN. In a real situation, the majority of tricks are proved by experiments instead of rigorous mathematical proofs. Nowadays, Parameters Server frame, based on delayed SGD algorithms, is the most popular learning frame. However, with the increasing number of workers, the burden of network would be unaffordable. Asynchronous FTRL and addressing L2 norm on server are two widely used tricks to solve this problem, but they are not rigorously proved. Hereafter, these two tricks will be abbreviated as asynch-FTRL and L2 norm trick. In this paper, we show the proof of Asynchronous Composite Objective MIrror Descent, abbr. asynch-COMID. Based on asynch-COMID, we establish the equivalence between asynchronous COMID and the above two tricks. Thus, the convergences of above two tricks are also proved. We fill these gaps between application and theory of above two tricks via asynch-COMID. Machine learning and stochastic optimization Training process in machine learning can be treated as solving the stochastic optimization problem. The objective functions are the mathematical expectation of loss functions which contain random variable. The random variables satisfy with known distribution. In a real situation, we use the frequency to approximate the product of probability density *Corresponding author density(x) and ∆x, as frequency histogram can roughly estimate the curve of probability density function. So, for a dataset, the above formula can be written as the following form: minE[g(X,w)](X ∼ certain distribution D) = min ∫ g(x,w)density(x)∆x